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Introduction 


The goal of this brief and accompanying Excel model is to provide school district and school staff 
with tools to help make more effective and efficient use of resources, particularly American Rescue 
Plan ESSER III funding. ESSER III funding offers districts and schools an opportunity to not only 
support their immediate needs of safely reopening schools to in-person instruction and addressing 
learning loss due to COVID-19-related interruptions to schooling, but also leverage opportunities 
for planning and restructuring for long-term improvement. An important component of this is to 
budget and use scarce resources strategically, that is, using data to identify student needs, establish 
performance goals, and respond with strategies and programs capable of meeting these goals that 
are both effective and cost-effective. What is the difference between effectiveness and cost- 
effectiveness? A strategy or program may be highly effective but also very costly, perhaps beyond 
the reach ofa district’s budget. Other alternatives may be available that are as effective, or nearly as 
effective, but cost significantly less. If the impact of one of more of these alternatives is sufficient for 
meeting performance goals, then any one of these lower cost options may be a better, more cost- 
effective choice. 


Cost-effectiveness (CE) analysis is a tool that may be used by districts and schools to compare the 
costs and effectiveness of two or more program alternatives with similar objectives. CE analysis 
requires both a detailed accounting of the costs of each alternative and a common outcome that 
may be compared across the alternatives under consideration. The product of a cost-effectiveness 
study is the cost-effectiveness ratio for each alternative. The cost-effectiveness ratio tells us the cost 
per unit of improvement, such as a one-point increase in scale scores, for each alternative 
intervention. For example, a district may be weighing five different elementary math interventions. 
A CE analysis will identify the per-student cost of each alternative, the expected average marginal 
unit of math achievement improvement per student, and the cost per unit of improvement. All other 
considerations being equal, the most desirable alternative is the one with the lowest cost- 
effectiveness ratio (i.e., the lowest cost per unit of improvement). 


Conducting a cost-effectiveness study involves the following five steps: 


» Step 1: Identifying viable alternatives. This step involves identifying the available alternatives 
for achieving a given outcome most appropriate for a given district or school’s context (e.g., 
multiple interventions for improving elementary math scores). 


» Step 2: Locating reliable impact studies for each alternative that use the same measure of 
effectiveness for each alternative (e.g., the change in scores on a specific math assessment—or 
equated assessments or measures if multiple assessments are used). 


» Step 3: Estimating the costs of each alternative. 


» Step 4: Calculating the cost-effectiveness ratio of each alternative. 


1 See Levin, H.M., McEwan, P.J., Belfield, C., Bowden, B.A., and Shand, R. (2018). Economic Evaluation in 
Education: Cost-Effectiveness and Benefit-Cost Analysis. Los Angeles, CA: Sage. 
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» Step 5: Selecting the most appropriate, cost-effective alternative that is both compatible with a 
district’s culture and needs, and offers sufficient improvement for meeting its performance 
goals. 


Determining Costs 


The preferred method for collecting program costs is the ingredients method recommended by the 
Center for Benefit-Cost Studies in Education.” Typical cost ingredients associated with educational 
interventions include personnel, materials, technology or other equipment, professional 
development, facilities, and potentially other resources. 


The purpose behind the ingredients approach is to account for the cost of all of the resources 
required to implement each alternative under consideration, irrespective of their source. By 
focusing on ingredients, this approach begins not with a budget, but with the details of the 
intervention and its resource requirements, such as the amount of staff time or instructional 
materials required. Only after systematically identifying all of the ingredients used in each of the 
strategy or program alternatives should the costs of these ingredients be measured (e.g., if the 
ingredient is 30 hours of teacher time, then the cost is 30 hours’ worth of salary and fringe 
benefits). Further, these costs should be calculated as incremental costs (i.e., taking into account 
resources utilized as part of the intervention beyond those supporting the current approach), also 
known as the “business as usual” approach. For example, if assessing alternative supplemental 
math interventions, costs should only include those of the intervention itself, not the costs of 
delivering the district’s adopted math curriculum. 


Evaluators and educators often believe that the best method of gathering cost information is to rely 
on available program budgets. However, for a number of reasons budgets are inadequate for 
determining costs. Budgets using state-approved accounting codes may not provide sufficient detail 
for isolating the specific costs of an intervention. The budgeted amount also may not equal what 
was actually spent on a strategy or intervention. Finally, most budgets may not include certain costs 
of an intervention, such as the cost of facilities used in the intervention, unless space is being rented 
or there is new construction. 


Because a CE analysis is typically undertaken before any alternatives are actually purchased and 
implemented, districts will likely need to estimate costs based on a description of program 
requirements provided by a program’s vendor, or based on prior experience of implementing 
similar programs, or on the experience of similar districts that have already implemented the 
program, or on some combination of all three approaches. No matter how costs are estimated, 
every effort should be made to ensure as complete and accurate estimates as possible for each 
alternative included in the analysis. 


2 See the Center for Benefit-Cost Studies of Education, University of Pennsylvania Graduate School of 
Education. https://www.cbcse.org/. 
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Determining Effectiveness 


The measure of effectiveness for the alternatives under consideration should closely reflect the 
main objective of the alternatives (e.g., a valid reading assessment for measuring the effects of 
alternative reading interventions). One of the constraints of the cost-effectiveness method is that it 
requires acommon and comparable outcome measure for each of the alternatives being studied. 
Therefore, cost-effectiveness analysis cannot be used to compare interventions with different goals; 
for example the cost-effectiveness of a reading intervention compared to that of a math 
intervention. 


Key to identifying and comparing alternative strategies and interventions are high-quality 
evaluations that provide accurate information on their effectiveness and their appropriateness for 
different types of students. While there may be occasions when a school district may be able to pilot 
and evaluate a program’s effectiveness on its own, in most cases districts will likely rely on the 
research of others to provide estimates of the effects or benefits. 


The effectiveness of educational strategies, programs, or interventions is determined through research 
studies aimed at measuring their “impact.” Impact refers to the effect the intervention has on student 
learning or other outcomes of interest. For example, how much a supplemental reading program 
increases students’ scores on a reading assessment. Impact studies may vary in quality. Some factors to 
consider include: 


» How many students were included in the study? Typically, the more the better. 


» What method was used for the study? The gold standard is randomized control trials where 
participating students are randomly assigned to a treatment group (they are taught using the 
program being studied) and a control group (they are not taught using the program of interest, 
but instead using business as usual materials). Quasi-experimental designs may also be found in 
higher quality studies. Quasi-experimental design studies do not use random assignment but use 
other means to identify treatment and control groups. Some examples of quasi-experimental 
design studies include regression discontinuity, interrupted time series, or natural experiments. 


» Was the study conducted internally by the developer/vendor of the product or by external 
researchers? Developers of educational programs or interventions often conduct their own 
impact studies to demonstrate effectiveness. It is generally preferable if an independent team of 
researchers conducts the study. However, if the results of the study were published in a peer- 
reviewed journal then the results are likely acceptable regardless of who conducted the study. 
Another check is if an independent study team subsequently conducted a similar study which 
showed similar results. 


Another factor to consider is whether the context of the districts, schools, and students 
participating in the study are similar to those of your district. A program that is effective for one 
group of students may not be similarly effective for another. Ideally, study results of the program or 
intervention show it is effective for students similar to those your district serves. 
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There are several sources of reliable impact studies districts may turn to, including those published 


in peer-reviewed academic journals, evaluation reports, or studies available on the What Works 


Clearinghouse website. 


» 


What Works Clearinghouse. Provided by the U.S. Department of Education’s Institute for 
Education Sciences (IES), the What Works Clearinghouse (WWC) provides full research reports 
and summaries of high-quality program evaluations. WWC not only presents evaluation findings 
but also rates whether or not the studies meet WWC research quality standards. 


Link: https://ies.ed.gov/ncee/wwc/ 


Evidence-Based School Finance Literature Reviews. One of the four primary methods for 
determining adequate funding for schools is the evidence-based approach. This approach makes 
use of an extensive literature review of evidence-supported programs and strategies to estimate 
the cost of an adequate education. A recent version of this literature review may be found in the 
report for a 2020 study in Wyoming (see review beginning on page 55). 


Link: http://picusodden.com/wp-content/uploads/2021/03/Wyoming-EB-Adequacy-Study- 
2020.pdf 


RAND Educational Program Evaluation. The RAND Corporation’s website features reports 
and briefs on their evaluations of a range of education program areas. 


Link: https://www.rand.org/topics /educational-program-evaluation.html 


Publications from the American Educational Research Association. Two of its journals 
regularly featuring program evaluations are Educational Evaluation and Policy Analysis and The 
American Educational Research Journal. Both are peer reviewed. 


Links: _https://journals.sagepub.com/description/EPA 


https://journals.sagepub.com/home/aer 


Educational Research and Evaluation. A peer reviewed journal that publishes articles on 
educational research and evaluation on a wide variety of education topics. It is international in 
scope. 


Link: https://www.tandfonline.com/toc/nere20/current 


Determining the Cost-Effectiveness Ratio 


Once the alternatives’ costs have been estimated and a common effectiveness measure identified, 


the cost-effectiveness ratios (CERs) may be calculated. This ratio is calculated as the cost divided by 


the effectiveness measure—in most cases the per-student cost of an alternative divided by the 


average improvement by student. For example, if a program costs $200,000 and yields 100 extra 


high school completers above and beyond what would be expected from a comparison group not 


participating in the program, the CER is $2,000 per extra completer. This ratio shows the cost of 


“buying” an extra completer. 
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The cost-effectiveness ratio equals: 


Cost of alternative 


Change in unit of effectiveness 


Lower CERs are generally preferred—if the program referenced above yields 200 extra completers, 
then the CER improves to $1,000 per extra completer. This ratio may be helpful because it can be 
easily related to the value of the program—specifically, whether it is worth spending $1,000 to 
$2,000 to “buy” an additional completer. 


CERs may also be tested for sensitivity to see how sensitive the results are to alternative modeling 
assumptions. Sensitivity testing may include examining the results if the best and worst cases are 
assumed, thereby placing upper and lower bounds on the results. Another type of sensitivity test is 
to vary key variables, or parameters in the model, such as per-student cost. Is the most cost- 
effective alternative still the same if per-student costs are $50 or $100 per student higher than 
estimated? 


Cost-Effectiveness Example 


This example of a CE analysis compares the cost-effectiveness of four hypothetical alternative 
elementary mathematics curricula. The analysis follows the study procedures described above— 
1) identifying viable alternatives; 2) locating reliable impact studies for each alternative that use 
the same measure of effectiveness for each alternative; 3) estimating the costs of each alternative; 
4) calculating the cost-effectiveness ratio; and 5) selecting the most appropriate, cost-effective 
alternative. The screen shots of analysis tables are taken from the companion Cost-Effectiveness 
Calculator. 


Step 1: Identifying viable alternatives 


For this example, we looked to the Institute for Education Sciences’ What Works Clearinghouse 
(WWC) for representative examples of elementary math curricula. Based on actual curricula 
reviewed by WWC, four hypothetical alternatives were developed representing a range of 
effectiveness levels. A school district searching for appropriate alternatives would likely have 
additional criteria for selecting alternatives based on scope and sequence across all school levels, 
the needs of its students, and its specific philosophy of math instruction. 


Step 2: Locating reliable impact studies for each alternative that use 
the same measure of effectiveness for each alternative 


The WWC not only did the leg-work of locating impact studies for each alternative, but also 
assessed the quality of the studies. Because WWC provides its own Improvement Index for all 
curricula reviewed on the website, this was used as the common measure of effectiveness. 
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Figure 1 shows the Effectiveness Value entered for each math curriculum alternative, the 
effectiveness measure selected for this analysis (shown in the “Select Type of Effectiveness Measure 
Used” box to the right), and the estimated number of student participants for each alternative. The 
WWC Improvement Index ranges from 2 index point for Alternative 4 to 11 index point for 


Alternative 1. 


Figure 1. Effectiveness measures and estimated participants 


Enter each alternative's name and effectiveness value below: 


Enter 
Effectiveness 
Value Select Type of Effectiveness Measure Used 
Alternative 1 |Math Curriculum A 11 WWC Improvement Index 
Alternative 2_ |Math Curriculum B 12 
Alternative 3. |Math Curriculum C 8 If "Other", what is the measure used? 
Alternative 4 |Math Curriculum D 2 


Enter the number of students or staff participating in each alternative below: 


Enter Number 
of Participants 


Alternative 1 |Math Curriculum A 347 
Alternative 2. |Math Curriculum B 277 
Alternative 3_ |Math Curriculum C 885 
Alternative 4 |Math Curriculum D 800 


Step 3: Estimating the costs of each alternative 


For the purpose of this example no attempt was made to estimate the actual costs of purchasing 
and implementing each alternative. Instead, hypothetical average costs for districts participating in 
the studies were constructed assuming an average-sized participating district for each alternative 
based on the total number of students participating in relevant studies for each alternative 
reported by WWC. Estimated costs for each alternative are presented in figure 2 below. Because 
cost-effectiveness is estimated based on the average expected gain per student, the cost amounts 
must also be on a per-student basis. Figure 2 shows that per-student costs range from $220.06 for 
Alternative 3 to $340.13 for Alternative 1. 
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Detailed costs of four elementary mathematics curricula 


Figure 2. 


Units: 


Price/Uni 


Personnel Costs 


Administrator salaries - central office Hours 20.0 $48.08 $961.54 Hours $480.80] Hours . 12.0 $48.08 $576.96] 
Administrator salaries - school Hours 90.0 $43.27 $3,894.23 Hours $3,245.25 Hours % 90.0 $43.27 $3,894.30) 
Clerical/support staff salaries Hours 180.0 $16.83 $3,028.85 ff Hours $1,683.00 ff Hours : 120.0 $16.83 $2,019.60 
Instructional coach, mentor, or lead teacher salaries Hours $0.00] Hours $5,041.08 { Hours i 384.0 $38.19 $14,664.96} 
Teacher salaries Hours 336.0 $38.19 $12,833.33 ff Hours $12,602.70 Hours i 960.0 $38.19 $36,662.40 
Subsitute teacher salaries Hours 224.0 $15.00 $3,360.00 Hours $2,640.00f| Hours . . 512.0 $15.00 $7,680.00) 
Other certified staff salaries $0.00) 
Other classified staff salaries $0.00 
Fringe benefit costs for certified staff % of Salaries 30.0% 96 of Salaries 30.0% — $6,410.95]§% of Salaries 30.0%  $18,096.63)§.% of Salaries 30.0% $16,739.59] 
Fringe benefit costs for classified staff % of Salaries 24.0% % of Salaries 24.0% $403.92] % of Salaries 24.0% $747.259§ % of Salaries 24.0% $484.70 
Fringe benefit costs for substitute teachers % of Salaries 15.0% 96 of Salaries 15.0% $396.00 f§% of Salaries 15.0% __$1,260.00)§% of Salaries $1,152.00 


Total Personnel Costs $83,874.51) 


Non-Personnel Costs 


Contract costs of consultants and external trainers 1.00 $25,000.00 $30,000.00 20,000.00 1.00 30,000.00 $30,000.00 
Technology hardware or equipment $0.00 
Technology software 1.00 25,000.00 $25,000.00 
Licensing or subscription fees 10.00 $5,000.00 3,000.00 17.00 3,500.00 $59,500.00 
Materials or supplies 1.00 $1,908.00 1.00 $16,620.00 1.00 5,310.00 1.00 4,800.00 $4,800.00} 
Professional development Teachers 14.00 $750.00 Teachers 11.00 $675.00 425.009] Teachers x 500.00 32,00 650.00 $20,800.00 
Food and refreshments $0.00 
All other costs $0.00} 

Total Non-Personnel Costs | $140,100.00] 

Total Costs | $223,974.51] 

Total Cost per Participant | $279.97] 


Step 4: Calculating the cost-effectiveness ratio 


Figure 3 below shows the resulting cost-effectiveness ratio, which is calculated automatically by the 
Cost-Effectiveness Calculator. The cost-effectiveness values range from $26.16 per index point 
improvement on the Improvement Index for Alternative 2 to $139.98 for Alternative 4. 


Figure 3. Cost-effectiveness ratios for the four mathematics curricula alternatives 


Alternative 1 


Math Curriculum A 


$30.92 


Alternative 2 


Math Curriculum B 


$26.16 


Alternative 3 


Math Curriculum C 


$27.51 


Alternative 4 


Math Curriculum D 


$139.98 


NOTE: The alternative with the lowest Cost-Effectiveness ratio (the lowest dollar amount shown 
because the CE Ration represents the cost per unit of change provides the lowest cost per unit of 
change among the alternatives under consideration. 
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Step 5: Selecting the most appropriate, cost-effective alternative 


Because in this example there are no other considerations other than selecting the most cost- 
effective option, our preferred math curriculum would be Alternative 2, which has the lowest cost 
per unit of Improvement Index gain. As noted above, districts may have other legitimate criteria for 
selecting an alternative that is less lower cost-effective but a better fit for the district. 
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